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1.  INTRODUCTION 


Prostate  cancer  is  the  second  leading  cause  of  cancer  death  and  the  most  common  cancer  detected  in  men  in 
the  United  States.  Radical  Prostatectomy  (RP)  is  a  surgical  procedure  with  a  goal  of  complete  cancer  resection. 
Open  radical  prostatectomies  are  associated  with  loss  of  blood  and  lengthy  recovery  times.  Laparoscopic  radical 
prostatectomies  (LRP)  have  a  very  steep  learning  curve,  lack  3-D  visualization  and  use  rigid,  stick-like 
instruments  which  hinder  the  dissection.  A  third  technique  has  recently  emerged;  laparoscopy  using  the  daVinci 
Surgical  System  (Intuitive  Surgical,  Sunnyvale,  CA).  The  surgical  robot  introduces  many  benefits,  including 
three-dimensional  visualization,  higher  magnification,  hand  tremor  elimination  and  refined  dexterity  by 
incorporating  wristed  instrumentation.  Robotic-assisted  laparoscopic  prostatectomy  (RALP)  strives  to  maximize 
tumor  resection  and  nerve  preservation  and  it  has  rapidly  become  the  preferred  surgical  approach.  Recent 
studies  have  showed  RALP  is  a  feasible  procedure  with  a  short  learning  curve,  limited  blood  loss,  less  post¬ 
operative  pain,  favorable  complication  rates,  and  short  hospital  stay.  The  lack  of  tactile  feedback  is  however, 
one  theoretical  disadvantage  that  has  been  raised  with  regards  to  robotic  surgery.  There  is  a  need  for  real-time 
intraoperative  guidance  in  laparoscopic  prostatectomies.  Hence  the  objective  of  this  proposal  is  to  develop  such 
a  capability.  Ultrasound  Elastography  (USE)  is  a  technology  which  can  be  integrated  with  the  robotic  probe. 
Elastography  is  ideal  as  a  technology  as  it  allows  for  real-time  acquisition  of  images  of  the  prostate  gland  and, 
similar  to  human  palpation,  allows  for  contact  based  interrogation  of  the  prostate’s  surface  [1].  USE  is 
affordable  and  minimally  invasive  and  can  also  be  miniaturized  and  incorporated  into  robotically  assisted 
prostatectomy.  USE  using  a  laparoscopic  ultrasound  probe  (LAPUS)  can  help  the  surgeon  visualize  the 
anatomy  of  the  prostate  gland,  identify  the  contours  of  the  cancerous  tumors  as  well  as  any  extra  capsular 
extension.  The  contribution  of  the  proposed  research  project  is  that  it  will  allow  for  real-time  intraoperative 
acquisition  of  images  of  the  prostate  gland  and  the  surrounding  tissues.  We  are  proposing  a  safe,  simple  and 
robust  technique  for  direct  interrogation  of  the  prostate  surface  aided  by  a  laparoscopic  US  probe.  USE  will  be  a 
valuable  tool  in  the  identification  of  cancerous  lesions  and  the  trajectory  of  cavernous  nerves  and  thus  improve 
the  chances  for  a  cancer-free,  nerve-sparing  outcome. 

2.  BODY 

This  report  will  address  each  of  the  tasks  we  proposed  to  accomplish  in  the  Statement  of  Work  during  the  first 
year  of  the  award,  as  outlined  below: 

Task  2.  System  Integration  and  Clinical  Evaluation  (months  1-12) 

2a.  Integration,  testing,  pre-operative  plan  (months  1-6).  We  proposed  an  integration  of  an  existing 
laparoscopic  ultrasound  probe  (Intuitive  Surgical,  Sunnyvale,  CA)  of  7.5  MHz  frequency  with  the  daVinci 
surgical  system.  We  can  report  the  integration  has  been  done  successfully  in  our  lab,  as  shown  in  the  publication 
submission  in  Appendix  1  (Billings,  S.,  Nishikant,  D.,  Kang,  H.  J.,  Taylor,  R.,  Boctor,  E.,  "System  for  robot- 
assisted  real-time  laparoscopic  ultrasound  elastography,"  Pending  submission  to  SPIE  Medical  Imaging  2012). 

Figure  1  shows  a  block  diagram  of  our  complete  system  for  robot-assisted  ultrasound  elastography.  The  daVinci 
S  surgical  robot  is  the  primary  component  of  our  system.  The  experimental  “Read/Write”  Research  Application 
Programming  Interface  (API)  provided  by  Intuitive  Surgical  Inc.  was  used  to  enable  robot  motion  to  be  controlled 
from  computer  in  addition  to  control  inputs  from  the  master  console.  The  daVinci  robot  manipulates  a  prototype 
version  of  a  robotically  articulated  laparoscopic  ultrasound  probe,  also  developed  by  Intuitive  Surgical  Inc., 
which  was  built  into  the  form  factor  of  a  standard  daVinci  tool.  The  ultrasound  probe  is  driven  by  a  Sonix  RP 
ultrasound  system  (Ultrasonix  Medical  Corp.,  Richmond  BC  Canada),  which  provides  an  Ultrasound  Research 
Interface  granting  access  to  pre-beam  formed  RF  data  from  the  ultrasound  probe.  The  RF  is  read  and  processed  in 
real  time  using  elastography  algorithms  developed  in  our  lab.  A  high-performance  external  NVIDIA  Tesla  series 
GPU,  which  connects  locally  to  the  Sonix  RP  system,  provides  sufficient  parallel  computing  power  to  generate 
strain  images  from  RF  data  in  real.  The  elastography  images  from  the  GPU  are  streamed  from  the  Sonix  RP 
system  to  a  network  port  on  the  system’s  central  workstation.  The  workstation  also  receives  a  parallel  stream  of 


conventional  B-mode  ultrasound  images  from  the  Sonix  RP  machine  directly.  The  workstation  sends  both 
image  streams  to  picture-in-picture  overlays  shown  in  the  stereo  display  of  the  daVinci  console  (Figure  2).  The 
picture  overlays  enable  the  surgeon  to  observe  the  ultrasound  and  elastography  image  feeds  in  real-time  without 
releasing  control  of  the  robot  arms  or  diverting  attention  away  from  the  task  at  hand. 
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Figure  1 .  Block  diagram  of  the  robot-assisted  system  for  real-time  ultrasound  elastography 


After  integration  the  system  was  tested  on  tissue  mimicking  phantoms.  An  automatic  palpation  motion  sequence 
was  also  implemented.  This  represents  in  fact  the  completion  of  task  3c.  Automatic  robotic-assisted  palpation 
(months  18-24).  The  assumption  was  that  automatic  pressure  application  will  produce  more  stable  and 
repeatable  images  and  thus  eliminate  inter-observer  variability  in  interpretation.  In  this  implementation  the 
tissue  is  autonomously  compressed  along  the  axial  direction  of  the  ultrasound  probe.  The  system  computer 
generates  a  sinusoidal  palpation  motion  that  is  overlaid  onto  motion  commands  from  the  master  manipulators. 
The  surgeon  retains  control  over  the  position  and  orientation  of  the  ultrasound  probe  but  the  computer  provides 
the  tremor-free,  precise  compression  motion  necessary  for  achieving  a  high  quality  strain  image.  Continual 
palpation  of  tissue  under  manual  control  is  a  tedious  task  imposing  a  large  cognitive  burden  and  demand  of 
focus.  By  relieving  this  burden  through  computer  assistance,  the  user  is  able  to  focus  on  more  important  tasks 
such  interpreting  real-time  imaging  information  and  conducting  surgery. 

The  results  are  reported  in  the  same  publication  attached  in  Appendix  1  (Billings,  S.,  Nishikant,  D.,  Kang,  H. 
J.,  Taylor,  R.,  Boctor,  E.,  "System  for  robot-assisted  real-time  laparoscopic  ultrasound  elastography,"  Pending 
submission  to  SPIE  Medical  Imaging  2012).  Figure  2  shows  a  view  of  the  daVinci  console  display.  The  overlay 
of  the  elastogram  shows  a  clear  differentiation  of  harder  and  softer  lesions. 
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Figure  2.  The  da  Vinci  console  display  showing  real-time  imaging  overlays  during  a  test  with  an  elasticity  phantom; 
the  elasticity  image  differentiates  between  two  lesions  of  varying  stiffness. 


Part  of  task  2a  was  to  work  with  Dr.  Allaf  and  our  clinical  team  to  put  together  a  plan  to  be  carried  into  the 
operating  room  for  testing  of  the  laparoscopic  ultrasound  probe  in  a  clinical  setting.  This  plan  would  have  been 
done  in  conjunction  with  task  2b:  IRB  approval  (months  1-6)  in  which  Institutional  Review  Board  approval 
for  a  small  (N=5-10)  clinical  study  using  the  LAPUS  probe  was  going  to  be  sought  and  the  subsequent  2c. 
Clinical  evaluation  (months  6-12).  Given  the  successful  system  integration,  the  implementation  of  automatic 
palpation  and  the  successful  testing  on  tissue  mimicking  phantoms,  our  team  felt  the  next  step  would  be  to  test 
the  system  on  ex-vivo  prostate  specimens.  This  comes  as  an  intermediary  step  and  a  natural  bridge  to  clinical 
testing.  In  the  mean  time,  Intuitive  Surgical  is  also  seeking  clinical  approval  for  the  prototype  LAPUS  we  used 
in  our  system  integration. 

We  are  working  on  an  Institutional  Review  Board  approval  for  a  medium  scale  study  (N=20-25)  on  ex-vivo 
excised  prostate  specimens.  The  first  objective  of  the  study  will  be  to  test  and  evaluate  the  LAPUS  probe.  The 
prostate  will  be  embedded  in  a  gel  phantom  and  it  will  be  scanned  using  the  LAPUS  probe.  The  phantom  will 
be  also  scanned  with  a  regular  ultrasound  probe  approved  for  clinical  use.  A  3D  Elastography  data  set  will  be 
collected  for  each  specimen  and  a  database  will  be  created.  A  box  will  be  build  (design  in  progress)  for  the 
specimen  phantom  with  enclosed  fiducials  which  will  allow  for  alignment  and  comparison  of  the  collected  data 
sets.  Additional  imaging  modalities  like  MRI  can  be  collected  and  used  for  comparison.  After  imaging,  the 
specimen  will  undergo  macro-  and  micro-  pathological  processing.  Histopathology  represents  the  gold  standard 
for  cancer  detection.  We  will  compare  our  results  with  the  pathological  findings. 


The  second  objective  of  the  study  will  be  to  prepare  for  clinical  testing  of  the  LAPUS  probe.  The  material 
surrounding  the  specimen  can  be  altered  so  the  phantom  will  mimic  the  approach  of  the  robotic-assisted 
laparoscopic  prostatectomy.  This  will  help  us  define  image  acquisition  points  and  planes.  We  have  to  ensure 
that  our  probe  is  able  to  maneuver  and  acquire  good  data  suitable  for  the  elastography  algorithm.  Additionally, 
our  goal  is  to  have  no  impact  on  the  outcome  of  the  surgical  procedure  and  also  not  to  extend  the  length  of  the 
surgery  more  than  10-15  minutes. 

We  have  experienced  delays  in  the  submission  of  the  IRB  due  to  institutional  issues.  JHMI  (Johns  Hopkins 
Medical  Institutions)  performs  hundreds  of  prostatectomies  every  year.  We  have  been  working  with  our  busy 
Pathology  department  and  labs  to  ensure  that  our  project  does  not  disrupt  the  day-to-day  work  flow  for 
pathological  processing.  The  creation  and  design  of  our  phantom  box  is  one  of  the  results  of  this  collaboration. 
It  will  assure  that  pathologists  can  process  the  specimens  faster  and  with  higher  accuracy,  and  also  (for  our 
study  only)  the  pathologists  will  be  able  to  create  a  3D  map  of  the  pathological  findings  which  can  be  aligned 
and  compared  to  the  Elastography  findings.  This  is  not  currently  the  standard  procedure  at  JHMI  so  an 
additional  effort  from  our  pathology  team  will  be  needed,  as  well  as  understanding  of  the  special  procedure  by 
the  JHMI  Review  Board.  The  IRB  will  be  submitted  in  the  next  month  and  we  anticipate  the  completion  of  the 
study  and  the  evaluation  of  the  data  to  be  done  before  the  end  of  the  second  year  of  the  award. 

Additional  efforts  have  been  made  to  improve  our  Elastography  algorithm.  The  robustness  of  the  Dynamic 
Programming  algorithm  has  been  improved  and  the  progress  is  detailed  in  a  scientific  article  which  has  been 
submitted  for  publication  -  Appendix  2  (Fleming,  I.,  Rivaz,  H.,  Boctor,  E.,  Hager,  G.  “Robust  Dynamic 
Programming  Method  for  Ultrasound  Elastography”,  Pending  submission  to  SPIE  Medical  Imaging  2012).  This 
progress  was  very  important  as  we  proceeded  in  extending  our  algorithm  to  3D,  enabling  better  visualization 
and  understanding  of  the  cancerous  lesions,  their  localization  and  their  extension  throughout  the  prostate.  A  3D 
Elastography  has  been  implemented;  we  are  in  the  process  of  evaluating  it  on  simulated  ultrasound  data  and  or 
tissue  mimicking  phantoms.  This  effort  is  a  step  forward  in  the  completion  of  Task  4a-c.  Laparoscopic 
Prostatectomy  with  Advanced  US  Imaging  Methods  (months  21-32). 


3.  KEY  RESEARCH  ACCOMPLISHMENTS 

•  Integration  of  an  existing  laparoscopic  ultrasound  probe  with  the  daVinci  surgical  system. 

•  Testing  of  the  new  integrated  system  on  tissue  mimicking  phantoms 

•  Automatic  robotic-assisted  palpation:  design,  implementation  and  testing 

•  Design  of  a  phantom  box  to  be  used  in  an  upcoming  study;  it  will  allow  for  better  registration  of 
imaging  data  sets  and  also  the  comparison  to  the  pathology  findings 

•  Improvement  of  the  robustness  of  our  software  and  the  extension  of  the  Elastography  algorithm  to  a  3D 
version 

4.  REPORTABLE  OUTCOMES 

1.  Fleming,  I.,  Rivaz,  H.,  Boctor,  E.,  Hager,  G.  “Robust  Dynamic  Programming  Method  for  Ultrasound 
Elastography”,  Pending  submission  to  SPIE  Medical  Imaging  2012. 

2.  Billings,  S.,  Nishikant,  D.,  Kang,  H.  J.,  Taylor,  R.,  Boctor,  E.,  "System  for  robot-assisted  real-time 
laparoscopic  ultrasound  elastography,"  Pending  submission  to  SPIE  Medical  Imaging  2012 

3.  CONCLUSION 

Laparoscopic  Ultrasound  probe  (Intuitive  Surgical,  Sunnyvale,  CA)  was  integrated  with  the  daVinci  surgical 
system  for  use  in  Robot-Assisted  Laparoscopic  Prostatectomy  (RALP).  Automatic  robotic-assisted  palpation 
(initially  planned  for  months  1 8-24)  was  completed.  Additional  testing  needed  before  proceeding  to  clinical  use 
is  in  progress.  The  new  integrated  system  has  been  tested  on  tissue  mimicking  phantoms.  IRB  submission  for 


clinical  study  is  still  in  preparatory  phase.  Advances  have  been  made  in  improving  the  robustness  of  our 
Elastography  algorithm  and  it  has  also  been  extended  to  a  3D  version. 
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ABSTRACT 

Surgical  robots  provide  many  advantages  for  surgery,  including  minimal  invasiveness,  precise  motion,  high  dexterity,  and  crisp 
stereovision.  One  limitation  of  current  robotic  procedures,  compared  to  open  surgery,  is  the  loss  of  haptic  information  for  such 
purposes  as  palpation,  which  can  be  very  important  in  minimally  invasive  tumor  resection.  Numerous  studies  have  reported  the  use  of 
real-time  ultrasound  elastography,  in  conjunction  with  conventional  B-rnode  ultrasound,  to  differentiate  malignant  from  benign 
lesions.  Several  groups  (including  our  own)  have  reported  integration  of  ultrasound  with  the  daVinci  robot,  and  ultrasound 
elastography  is  a  very  promising  image  guidance  method  for  robot-assisted  procedures  that  will  further  enable  the  role  of  robots  in 
interventions  where  precise  knowledge  of  sub-surface  anatomical  features  is  cmcial.  In  this  paper,  we  present  a  novel  robot-assisted 
real-time  ultrasound  elastography  system  for  minimally  invasive  robot-assisted  interventions.  Our  system  combines  a  daVinci  surgical 
robot  with  an  experimental  software  interface,  a  robotically  articulated  laparoscopic  ultrasound  probe,  and  our  GPU-based 
elastography  system.  Elastography  and  B-rnode  ultrasound  images  are  displayed  as  picture-in-picture  overlays  in  the  daVinci  console. 
Our  system  minimizes  dependence  on  human  performance  factors  by  incorporating  computer-assisted  motion  control  that 
automatically  generates  the  tissue  palpation  required  for  elastography  imaging,  while  leaving  high-level  control  in  the  hands  of  the 
user.  In  addition  to  ensuring  consistent  strain  imaging,  the  elastography  assistance  mode  avoids  the  cognitive  burden  of  tedious 
manual  palpation.  Preliminary  tests  of  the  system  with  an  elasticity  phantom  demonstrate  the  ability  to  differentiate  simulated  lesions 
of  varied  stiffness  and  clearly  delineate  lesion  boundaries. 

DESCRIPTION  OF  PurPOSE  /  NEW  WORK  PRESENTED 

Surgical  robots  provide  many  advantages  for  surgery,  including  minimal  invasiveness,  precise  motion,  high  dexterity,  and  crisp 
stereovision.  One  limitation  of  current  robotic  procedures,  compared  to  open  surgery,  is  the  loss  of  haptic  information  for  such 
purposes  as  palpation,  which  can  be  very  important  in  minimally  invasive  tumor  resection.  Numerous  studies  have  reported  the  use  of 
real-time  ultrasound  elastography,  in  conjunction  with  conventional  B-mode  ultrasound,  to  differentiate  malignant  from  benign  lesions 
in  prostate,  breast,  pancreas,  lymph  nodes,  and  thyroid.11'2'3'4,51  Ultrasound  elastography  is  thus  a  very  promising  image  guidance 
method  for  robot-assisted  procedures  that  will  further  enable  the  role  of  robots  in  interventions  where  precise  knowledge  of  hidden 
anatomical  features  is  crucial.  Several  groups  (including  our  own)  have  reported  integration  of  ultrasound  with  the  daVinci  robot.16'7,81 
In  this  paper,  we  present  a  novel  robot-assisted  real-time  ultrasound  elastography  system  for  minimally  invasive  robot-assisted 
interventions. 


METHODS 

In  general,  cancerous  tissue  has  higher  cell  density  than  surrounding  healthy  tissue,  which  leads  to  elevated  tumor  stiffness  that  can  be 
visualized  using  elastography  techniques.  As  first  described  by  Ophir  et  al., 191  the  principle  of  ultrasound  elastography  is  to  estimate 
tissue  stiffness  from  measurement  of  tissue  strain  induced  by  an  applied  force.  This  is  accomplished  by  measuring  relative  tissue 
displacements  between  ultrasound  image  pairs  under  different  states  of  induced  compression.  Because  ultrasound  elastography 
algorithms  assume  an  axial  compression,  strain  image  quality  may  degrade  when  non-axial  motion  or  probe  rotation  occurs  during 
compression.  Axial  compression  is  typically  on  the  order  of  only  a  millimeter  or  two,  and  even  small  amounts  of  unwanted  motion 
may  affect  image  quality.  Due  to  the  variability  in  manual  compression,  image  quality  for  free-hand  elastography  is  highly  dependent 
on  practitioner  skill  and  experience. 

The  quality  of  elastography  imaging  as  a  function  of  human  performance  is  minimized  by  our  robot-assisted  elastography  system 
through  computer-integrated  motion  control  for  tissue  compression,  whereby  tissue  is  autonomously  compressed  along  the  direction 
axial  to  the  ultrasound  probe.  The  system  computer  provides  assistive  control  of  robot  motion  by  generating  a  sinusoidal  palpation 
motion  that  is  overlaid  onto  motion  commands  from  the  master  manipulators  (Figure  1).  By  this  method,  the  surgeon  retains  overall 
control  of  the  ultrasound  probe  position  and  orientation,  while  the  computer  controls  the  finer  points  of  tissue  compression.  Because 
the  computer  has  accurate  knowledge  of  the  ultrasound  probe  position  and  is  not  subject  to  motion  errors  like  a  human  operator, 
consistent  tissue  compression  in  the  precise  axial  direction  is  ensured. 

The  elastography  assistance  mode  improves  the  precision  and  consistency  of  tissue  compression  beyond  what  can  be  achieved  by 
manually  controlled  teleoperation  while  also  reducing  the  cognitive  burden  for  the  user.  Continual  palpation  of  tissue  under  manual 
control  is  a  tedious  task  imposing  a  large  cognitive  burden  and  demand  of  focus.  By  relieving  this  burden  through  computer 
assistance,  the  user  is  able  to  focus  on  more  important  tasks  such  interpreting  real-time  imaging  information  and  conducting  surgery. 


daVinci  Console 


Follow 

Mode 


daVinci  Master 


Slave  Motion 
Commands 


Sonix  RP 


daVinci 

Slave 


Hand  +  Palpate 
Motion 


PIP  Image 
Overlay 


Palpate  Motion 
Overlay  f\f\ I 


Palpate 

Controller 


3D  User 
Interface 


SAW 

Workstation 


B-Mode 

Ultrasound 


Elastography 


Figure  3.  Block  diagram  of  our  robot-assisted  system  for  real-time  ultrasound  elastography. 

Figure  1  shows  a  block  diagram  of  our  complete  system  for  robot-assisted  ultrasound  elastography.  At  the  heart  of  the  system  is  the 
daVinci  S  surgical  robot  (Intuitive  Surgical  Inc.,  Sunnyvale,  CA).  Computer  integrated  control  of  robot  motion  is  facilitated  by  an 
experimental  “Read/Write”  Research  Application  Programming  Interface  (API)  provided  by  Intuitive  Surgical  Inc.,  which  enables 
robot  motion  to  be  controlled  from  computer  in  addition  to  control  inputs  from  the  master  console,  as  discussed  below.  The  daVinci 
robot  manipulates  a  prototype  version  of  a  robotically  articulated  laparoscopic  ultrasound  probe  (Figure  2a),  also  developed  by 
Intuitive  Surgical  Inc.,  which  was  built  into  the  form  factor  of  a  standard  daVinci  tool.  The  ultrasound  probe  has  a  linear  array 
transducer  (Gore,  Newark  DE)  with  128  elements  that  measures  50mm  in  length  with  a  probe  diameter  of  approximately  9mm.  The 
ultrasound  probe  is  driven  by  a  Sonix  RP  ultrasound  system  (Ultrasonix  Medical  Corp.,  Richmond  BC  Canada),  which  provides  an 
Ultrasound  Research  Interface  granting  access  to  pre-beam  formed  RF  data  from  the  ultrasound  probe.  Access  to  this  RF  data  is 
critical  to  reaching  optimal  performance  from  elastography  algorithms.  The  RF  data  is  processed  by  a  high-performance  external 
NVIDIA  Tesla  series  GPU,  which  connects  locally  to  the  Sonix  RP  system.  The  GPU  provides  sufficient  parallel  computing  power  to 
generate  strain  images  from  RF  data  in  real  time  using  an  elastography  algorithm  based  on  normalized  cross-correlation.1101  The 
elastography  images  from  the  GPU  are  streamed  from  the  Sonix  RP  system  to  a  network  port  on  the  system’s  central  workstation.  The 
workstation  also  receives  a  parallel  stream  of  conventional  B-mode  ultrasound  images  from  the  Sonix  RP  machine  directly.  The 
workstation  sends  both  image  streams  to  picture-in-picture  overlays  shown  in  the  stereo  display  of  the  daVinci  console.  The  picture 
overlays  enable  the  surgeon  to  observe  the  ultrasound  and  elastography  image  feeds  in  real-time  without  releasing  control  of  the  robot 
arms  or  diverting  attention  away  from  the  task  at  hand. 

In  collaboration  with  Intuitive  Surgical,  the  Engineering  Research  Center  for  Computer-Integrated  Surgical  Systems  and  Technology 
(CISST  ERC)  at  Johns  Hopkins  University  has  developed  an  open-source  software  framework  for  medical  robotics  and  computer 
assisted  surgical  systems  research,  which  we  call  Surgical  Assistant  Workstation  (SAW).111’12'11'141  This  open-source  framework  is  an 
extension  of  the  CISST  software  libraries  developed  at  Johns  Hopkins  to  enable  rapid  application  development  by  providing 
capabilities  including  basic  mathematics  and  numerical  routines,  thread  execution  management,  inter-process  communication, 
construction  and  control  of  video  pipelines,  and  many  other  things.11516'17,181  Although  the  software  components  providing  its  basic 
capabilities  are  all  open-source,  SAW  is  designed  to  be  compatible  with  proprietary  modules  through  module  wrappers  using  well- 
defined  interface  protocols.  In  particular,  we  have  developed  SAW  wrappers  that  provide  the  ability  to  interface  with  the  read-only 
and  read-write  research  interfaces  of  the  daVinci  robot.  The  capabilities  of  the  Read-Only  daVinci  Research  Interface  include  the 
ability  to  query  the  state  of  the  master  and  slave  robot  arms  and  of  user  console  events.1141  The  ReadAVrite  daVinci  Research  Interface 
extends  these  features  with  the  capability  to  use  software  to  command  motions  of  the  master  and  slave  robot  arms  and  to  trigger  user 
events.  One  such  feature  allows  us  to  superimpose  an  externally  computed  motion  onto  motion  inputs  from  the  master  manipulators. 
We  use  this  capability  in  our  system  to  create  computer  generated  motion  overlays  for  tissue  palpation.  Another  function  of  SAW 
allows  us  to  use  a  master  arm  controller  effectively  as  a  3D  mouse.  We  call  this  masters-as-mice  mode.  We  use  this  mode  in  our 
system  to  build  an  interactive  environment  for  the  daVinci  operator  to  control  and  interact  with  video  overlays. 

The  SAW  software  package  is  implemented  on  the  central  workstation  of  our  system,  which  centralizes  processing  for  user  interaction 
with  image  overlays,  acquisition  of  real-time  image  streams,  and  implementation  of  the  control  loop  generating  assistive  motion  for 
tissue  palpation.  A  command  terminal  on  the  central  workstation  allows  the  user  to  set  the  amplitude  and  frequency  of  tissue 
palpation.  These  settings  may  be  updated  in  real-time.  The  terminal  also  allows  palpation  to  be  activated/deactivated  at  will.  The  user 
interacts  with  the  image  overlays  from  the  daVinci  console  using  the  SAW  masters-as-mice  mode,  which,  for  the  daVinci  system,  is 
activated  by  engaging  the  clutch  foot  pedal  followed  by  a  double-pinch  of  both  master  manipulators  in  unison.  The  masters-as-mice 
mode  activates  an  interactive  menu  environment  in  which  the  daVinci  master  controllers  manipulate  virtual  cursors  in  the  daVinci 


console’s  stereo  display  (Figure  4a).  Menu  selections  are  made  by  moving  a  cursor  to  a  menu  icon  and  pinching  the  corresponding 
master  controller.  The  elastography  and  B-mode  ultrasound  image  overlays  are  activated  by  selecting  the  appropriate  menu  icon  with 
a  virtual  cursor  (pinching  master  controller  selects).  The  image  overlays  are  resized  and  repositioned  in  the  daVinci  display  by 
dragging  them  with  a  cursor.  Releasing  the  clutch  pedal  returns  the  master  manipulators  to  control  of  the  slave  arms  and  maintains  the 
image  overlays  in  the  user’s  field  of  view. 


Results 

Preliminary  tests  of  the  system  have  been  conducted  with  a  CIRS  Model  049  Elasticity  QA  Phantom,  which  has  simulated  lesions  of 
different  calibrated  stiffness.  Figure  4b  presents  a  snap-shot  of  the  daVinci  display  taken  during  phantom  testing  with  image  overlays 
displaying  the  real-time  imaging  results.  The  elastography  image  in  this  figure  shows  clear  difference  in  contrast  between  a  soft  and 
hard  lesion,  with  the  hard  lesion  appearing  darker  in  the  image.  This  distinction  cannot  be  made  from  the  B-mode  ultrasound  image  in 
this  figure.  The  elastography  image  also  establishes  clear  delineation  of  lesion  boundaries,  which  would  not  be  discernible  from  B- 
mode  ultrasound  in  the  case  of  isoechoic  lesions.  These  images  were  recorded  with  tissue  palpation  set  to  1mm  amplitude  and  1Hz 
frequency. 


CONCLUSION 

We  have  successfully  implemented  a  robot-assisted  system  for  minimally  invasive,  real-time  ultrasound  elastography.  Our  system 
provides  an  improvement  over  manual  elastography  techniques  by  unifying  motion  commands  from  a  user  with  the  precision  and 
accuracy  of  computer-assisted  motion  control  to  ensure  consistent  and  precise  tissue  strain.  Our  approach  effectively  reduces  the 
cognitive  load  of  the  human  operator  while  maintaining  a  user’s  control  of  the  procedure.  Preliminary  tests  using  an  elasticity  phantom 
demonstrate  the  system’s  capability  to  generate  strain  images  in  real-time  that  can  be  used  to  delineate  simulated  lesion  boundaries 
and  differentiate  lesions  of  varying  stiffness. 
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Figure  4.  a)  Interactive  menu  environment  for  displaying  image  overlays  in  the  daVinci  console  display,  showing  an  active  menu  with 
two  virtual  mice  corresponding  to  the  left  and  right  master  manipulators  and  two  picture-in-picture  image  overlays  (picture  taken  of  a 
patient  side  monitor),  b)  View  of  the  daVinci  console  display  during  a  test  with  an  elasticity  phantom;  the  elastography  image  overlay 
differentiates  lesions  of  different  stiffness. 
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Robust  Dynamic  Programming  Method  for 
Ultrasound  Elastography 


Ioana  Fleming,  Hassan  Rivaz,  Emad  Boctor,  and  Gregory  Hager 

Abstract.  Ultrasound  elastography  is  an  imaging  technology  which  can 
detect  differences  in  tissue  stiffness  based  on  tissue  deformation.  Initially 
introduced  more  than  two  decades  ago,  ultrasound  elastography  is  re¬ 
cently  establishing  itself  as  a  valuable  tool  for  diagnostic  and  monitor¬ 
ing.  For  clinical  successful  use  in  cancer  diagnosis  and  monitoring  the 
method  should  be  robust  to  sources  of  decorrelation  between  ultrasound 
images.  A  regularized  Dynamic  Programming  (DP)  approach  was  used 
for  displacement  estimation  in  compressed  tissue.  In  the  Analytic  Min¬ 
imization  (AM)  extension  of  DP,  integer  displacements  are  calculated 
just  for  one  RF-line,  and  are  then  propagated  lateraly  throughout  the 
entire  image.  This  makes  the  seed  RF-line  very  important;  faulty  seed 
lines  could  propagate  erroneous  displacement  values  throughout  the  im¬ 
age  resulting  in  the  appearance  of  false  lesions.  In  this  paper  we  analyze 
the  robustness  of  this  method  in  free-hand  palpation  of  laboratory  tissue 
phantoms.  We  are  proposing  an  update  to  the  algorithm  which  includes 
detection  of  faulty  DP  RF-lines  in  decorrelated  areas  and  solutions  for 
improving  the  robustness  of  the  DP  approach,  while  maintaining  real¬ 
time  computation  of  strain  images.  Axial  integer  displacement  is  ob¬ 
tained  on  each  RF-line  individually  using  DP  optimization.  The  values 
are  then  compared  among  adjacent  lines  and  several  metrics  are  consid¬ 
ered  which  could  detect  faulty  RF-lines.  We  are  considering  the  use  of 
filters  to  enforce  smoothness  and  similarity  among  neighboring  RF-lines. 
The  method  is  tested  and  the  results  are  compared  between  computer 
simulated  and  laboratory  tissue  phantoms. 


1  Background  and  Rationale 

Ultrasound  Elastography  is  an  imaging  technology  which  can  detect  differences 
in  tissue  stiffness  based  on  tissue  deformation.  Our  work  in  this  paper  focuses 
on  real-time  quasi-static  elastography.  The  tissue  is  compressed  and  relaxed  in  a 
continuous  free-hand  motion  and  ultrasound  images  are  simultaneously  acquired. 
This  method  is  easy  to  use  and  also  cheap  as  it  requires  no  extra  hardware.  This 
makes  it  particularly  appealing  for  medical  imaging  applications;  diagnosis  or 
monitoring  could  be  done  at  the  patient’s  bed  side.  There  are  two  hurdles  which 
once  resolved  would  ensure  success  for  clinical  use:  real-time  computation  of 
strain  images  and  dealing  with  the  potential  for  global  and  local  decorrelation 
between  pre-  and  post-compression  ultrasound  images.  Various  sources  of  decor¬ 
relation  are  affecting  the  computation  of  strain  images  in  in-vivo  data,  such  as 
incoherent  fluid  (blood)  motion,  out-of-plane  motion  of  structures  within  one 
image  due  to  transducer  or  respiratory  motion,  subsample  speckle  motion,  and 
a  high  degree  of  compression. 


Rivaz  et  al  (2008)  initially  proposed  optimizing  a  recursive  regularized  cost 
function  using  Dynamic  Programming  (DP).  The  method  resulted  in  integer 
values  for  axial  displacement  and  subsample  displacement  could  also  be  achieved 
but  at  a  high  computational  cost.  Rivaz  et  al  (2011)  refined  the  method  further 
using  a  ID  and  2D  Analitic  Minimization  (AM)  of  the  cost  function.  It  takes  the 
integer  displacement  of  a  single  radio  frequency  (RF)-line  from  DP  and  produces 
the  subsample  axial  and  lateral  displacement  fields  for  the  entire  image. 

In  this  work  we  analyze  the  robustness  of  the  AM2D  method.  Since  this 
method  is  based  on  computing  integer  displacements  using  DP  on  only  one  RF- 
line,  it  is  crucial  that  we  choose  a  starting  line  with  little  or  no  decorrelation 
between  the  two  images.  Here  we  present  a  method  for  identifying  faulty  DP 
lines  and  reducing  their  potential  to  generate  artifact  lesions. 


2  2D  AM  subpixel  displacement  estimation 


Here  we  briefly  review  the  2D  AM  method  in  which  2D  integer  displacements 
are  first  obtained  using  DP  on  a  single  RF-line  and  are  then  used  to  produce  2D 
subsample  displacements  for  the  entire  image. 


1.  Calculate  integer  axial  and  lateral  displacements  of  one  seed  RF-line  using 
DP  (Rivaz  et  al,  2008).  Calculate  an  initial  subsample  estimate  using  linear 
interpolation  of  the  integer  displacements. 

Let  I\  and  I2  be  two  ultrasound  images  acquired  before  and  after  defor¬ 
mation.  Let  m  be  the  number  of  RF-lines.  Each  signal  is  sampled  at  i  = 
1,  2  •  •  •  m.  A  regularized  cost  function  is  generated  combining  the  prior  of 
displacement  continuity  (regularization  term)  and  an  amplitude  similarity 
term.  The  displacement  continuity  term  for  line  j  can  be  written  as: 

Rj  (u?  dj,  &i—  1 ,  li— 1)  —  Q.a  (cti  &i  —  1 )  "P  Oil  (Ji  li  —  l)  (1) 


where  aa  and  ai  are  axial  and  lateral  regularization  weights  respectively. 
The  regularized  cost  function  at  the  ith  sample  of  the  jth  A-line  becomes: 
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where  w  is  a  regularization  weight  for  smoothness;  da  and  di  are  temporary 
axial  and  lateral  displacements  that  are  varied  in  order  to  minimize  eqn.  (2). 

2.  Calculate  subsample  axial  and  lateral  displacements  for  the  seed  RF-line 
using  2D  AM  (below).  They  will  be  added  to  the  initial  integer  estimates. 

3.  Propagate  the  solution  of  the  seed  RF-line  to  the  left  and  right,  using  the 
displacement  of  the  previous  line  as  initial  estimate. 

The  aim  is  to  calculate  Aai  and  AU  such  that  the  duple  (ai  +  Aai ,  k  +  AU) 
gives  the  axial  and  lateral  displacements  at  the  sample  i.  The  regularized 
cost  function  becomes:  where  the  index  j  was  dropped  for  the  jth  RF-line 
and  litj- 1  is  the  lateral  displacement  of  the  previous  RF-line  (except  for  the 


Fig.  1.  Strain  images  from  breast  phantom  freehand  palpation  using  DP  AM2D 
method.  Note  most  artifacts  occur  in  the  upper  part  of  the  phantom.  Only  1  (one) 
faulty  line  (line  #476)  produced  an  artifact  which  obscured  the  actual  lesion 


seed,  line  where  h,j- i  =  Z».  a,  /3a  and  f3l  are  regularization  terms  which  ensure 
continuity  in  displacements  with  respect  to  the  top  (axial  a),  and  the  top 
and  left/right  (lateral  f3a  and  Pt).  If  the  displacement  of  the  previous  line  is 
not  accurate,  it  will  affect  the  displacement  of 

C j  (Z\ or , . . . ,  Adm ,  Al\ , . ..,  Alm) 

=  -  h(i  +  a-i  +  Adi,j  +  k  +  Ali)]2 

+a  (at  +  Am  -  di- 1  -  Adi- 1)  (3) 
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the  next  line  through  the  last  term  in  the  right-hand  side  of  (3). 

3  Experimental  Design  and  Results 

For  experimental  evaluation  we  palpated  a  breast  elastography  phantom  with 
a  10mm  cf>  lesion  and  three  times  stiffer  than  the  background.  RF  data  was  ac¬ 
quired  with  a  7.27MHz  linear  array  at  a  sampling  rate  of  40MHz.  We  evaluated 
the  percentage  of  seed  RF-lines  with  wrong  DP  displacement  estimations.  The 
DP  displacement  computation  uses  a  smoothness  regularization  parameter  w 
(2)  which  should  prevent  regions  with  high  local  decorrelation  from  introducing 
errors  in  displacement  estimation,  but  if  chosen  too  large  would  result  in  over¬ 
smoothing.  We  used  3  (three)  values  for  w:  0.15,  0.3  and  0.6.  The  percentage  of 
faulty  seed  lines  was  38.8,  30.43,  and  74.4.  Here  we  report  on  the  w  =  0.3  case. 

Many  of  the  artifacts  created  by  the  erroneous  displacement  estimation  were 
very  small  (fig.  1  a)  and  localized  at  the  top  of  the  image.  Some  artifacts  were 
large,  sometimes  propagating  through  a  big  part  of  the  image  and,  in  only  one 
case,  the  resulting  artifact  lesion  obscured  the  real  lesion  (fig.  1  d).  The  distri¬ 
bution  of  the  faulty  lines  across  the  image  is  plotted  in  (fig.  2).  The  lines  appear 
to  align  with  peaks  and  nadirs  in  mean  displacement;  this  could  be  a  potential 
indicator  for  faulty  lines. 

The  change  in  mean  displacement  for  each  RF-line  could  be  a  metric  which 
would  be  able  to  separate  faulty  RF-lines.  Sometimes  a  group  of  adjacent  lines 


Fig.  2.  The  distribution  of  faulty  lines  (blue)  and  the  corresponding  change  in  mean 
pixel  displacement.  The  change  is  calculated  vs.  2  adjacent  right  and  left  RF-lines  (red) 
and  also  vs.  4  adjacent  lines(green)) 


Fig.  3.  Initial  axial  integer  displacement  estimation  (a),  after  median  5x5  filter  (b)  and 
after  majority  vote  5x5  filter  (c) 


can  all  be  faulty,  so  we  computed  this  metric  in  2  ways: 


Adj  =  E^Kj} 
Ad*  =  Ejf  {<*«} 
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(4) 

where  a,^  is  the  integer  axial  displacement  of  pixel  i  on  the  RF-line  j.  The 
expected  value  of  Adj  should  be  very  close  to  0.5,  and  0.25  for  Adj  . 

There  are  202  RF-lines  with  0.495  <  Adj  <  0.505  compared  to  145  faulty 
RF-lines.  Only  71.71%  of  faulty  RF-lines  (104)  are  picked  out  as  true  positives. 
For  detection  of  all  faulty  lines  one  would  need  to  adjust  the  threshold  to  0.5002 
and  0.4998,  which  also  leads  to  many  false  positives:  452  RF-lines  out  of  the 
total  of  477  (94.75%)  would  be  deemed  faulty.  Similarly  with  Adj:  96.2%  of 
lines  would  be  found  faulty  in  order  to  select  all  the  true  faulty  RF-lines.  78.62% 
of  lines  are  selected  as  true  positives  when  248  lines  (52%)  of  all  lines  are  found 
positive  (0.248  <  Adj  <  0.252).  The  large  number  of  false  positives  makes 
this  method  alone  not  a  very  good  predictor  of  faulty  lines.  It  is  however  clear 
faulty  RF-lines  could  be  adjusted  after  DP  displacement  computation  in  order 
to  obtain  a  smaller  change  in  mean  displacement.  We  implemented  and  tested 
median  filters  as  well  as  a  majority  vote  approach.  Results  are  summarized  in 
fig.  3.  Multiple  filter  passes  will  result  in  further  decreases  in  faulty  seed  lines. 
Note:  This  work  has  not  been  submitted  for  publication  to  any  other  venue. 


